Search for: All records

This paper presents the theoretical foundation, practical implementation, and empirical evaluation of a glove for interaction with 3-D virtual environments. At the dawn of the “Spatial Computing Era”, where users continuously interact with 3-D Virtual and Augmented Reality environments, the need for a practical and intuitive interaction system that can efficiently engage 3-D elements is becoming pressing. Over the last few decades, there have been attempts to provide such an interaction mechanism using a glove. However, glove systems are currently not in widespread use due to their high cost and, we propose, due to their inability to sustain high levels of performance under certain situations. Performance deterioration has been observed due to the distortion of the local magnetic field caused by ordinary ferromagnetic objects present near the glove’s operating space. There are several areas where reliable hand-tracking gloves could provide a next generation of improved solutions, such as American Sign Language training and automatic translation to text and training and evaluation for activities that require high motor skills in the hands (e.g., playing some musical instruments, training of surgeons, etc.). While the use of a hand-tracking glove toward these goals seems intuitive, some of the currently available glove systems may not meet the accuracy and reliability levels required for those use cases. This paper describes our concept of an interaction glove instrumented with miniature magnetic, angular rate, and gravity (MARG) sensors and aided by a single camera. The camera used is an off-the-shelf red, green, and blue–depth (RGB-D) camera. We describe a proof-of-concept implementation of the system using our custom “GMVDK” orientation estimation algorithm. This paper also describes the glove’s empirical evaluation with human-subject performance tests. The results show that the prototype glove, using the GMVDK algorithm, is able to operate without performance losses, even in magnetically distorted environments. 

While the availability of low-cost micro electro-mechanical systems (MEMS) accelerometers, gyroscopes, and magnetometers initially seemed to promise the possibility of using them to easily track the position and orientation of virtually any object that they could be attached to, this promise has not yet been fulfilled. Navigation-grade accelerometers and gyroscopes have long been the basis for tracking ships and aircraft, but the signals from low-cost MEMS accelerometers and gyroscopes are still orders of magnitude poorer in quality (e.g., bias stability). Therefore, the applications of MEMS inertial measurement units (IMUs), containing tri-axial accelerometers and gyroscopes, are currently not as extensive as they were expected to be. Even the addition of MEMS tri-axial magnetometers, to conform magnetic, angular rate, and gravity (MARG) sensor modules, has not fully overcome the challenges involved in using these modules for long-term orientation estimation, which would be of great benefit for the tracking of human–computer hand-held controllers or tracking of Internet-Of-Things (IoT) devices. Here, we present an algorithm, GMVDμK (or simply GMVDK), that aims at taking full advantage of all the signals available from a MARG module to robustly estimate its orientation, while preventing damaging overcorrections, within the context of a human–computer interaction application. Through experimental comparison, we show that GMVDK is more robust to magnetic disturbances than three other MARG orientation estimation algorithms in representative trials. 

In this paper, we present the FIU MARG Dataset (FIUMARGDB) of signals from the tri-axial accelerometer, gyroscope, and magnetometer contained in a low-cost miniature magnetic–angular rate–gravity (MARG) sensor module (also known as magnetic inertial measurement unit, MIMU) for the evaluation of MARG orientation estimation algorithms. The dataset contains 30 files resulting from different volunteer subjects executing manipulations of the MARG in areas with and without magnetic distortion. Each file also contains reference (“ground truth”) MARG orientations (as quaternions) determined by an optical motion capture system during the recording of the MARG signals. The creation of FIUMARGDB responds to the increasing need for the objective comparison of the performance of MARG orientation estimation algorithms, using the same inputs (accelerometer, gyroscope, and magnetometer signals) recorded under varied circumstances, as MARG modules hold great promise for human motion tracking applications. This dataset specifically addresses the need to study and manage the degradation of orientation estimates that occur when MARGs operate in regions with known magnetic field distortions. To our knowledge, no other dataset with these characteristics is currently available. FIUMARGDB can be accessed through the URL indicated in the conclusions section. It is our hope that the availability of this dataset will lead to the development of orientation estimation algorithms that are more resilient to magnetic distortions, for the benefit of fields as diverse as human–computer interaction, kinesiology, motor rehabilitation, etc. 

Abstract Alzheimer’s disease (AD) is a neurogenerative condition characterized by sharp cognitive decline with no confirmed effective treatment or cure. This makes it critically important to identify the symptoms of Alzheimer’s disease in its early stages before significant cognitive deterioration has taken hold and even before any brain morphology and neuropathology are noticeable. In this study, five different multimodal deep neural networks (MDNN), with different architectures, in search of an optimal model for predicting the cognitive test scores for the Mini-Mental State Examination (MMSE) and the modified Alzheimer’s Disease Assessment Scale (ADAS-CoG13) over a span of 60 months (5 years). The multimodal data utilized to train and test the proposed models were obtained from the Alzheimer’s Disease Neuroimaging Initiative study and includes cerebrospinal fluid (CSF) levels of tau and beta-amyloid, structural measures from magnetic resonance imaging (MRI), functional and metabolic measures from positron emission tomography (PET), and cognitive scores from the neuropsychological tests (Cog). The models developed herein delve into two main issues: (1) application merits of single-task vs. multitask for predicting future cognitive scores and (2) whether time-varying input data are better suited than specific timepoints for optimizing prediction results. This model yields a high of 90.27% (SD = 1.36) prediction accuracy (correlation) at 6 months after the initial visit to a lower 79.91% (SD = 8.84) prediction accuracy at 60 months. The analysis provided is comprehensive as it determines the predictions at all other timepoints and all MDNN models include converters in the CN and MCI groups (CNc, MCIc) and all the unstable groups in the CN and MCI groups (CNun and MCIun) that reverted to CN from MCI and to MCI from AD, so as not to bias the results. The results show that the best performance is achieved by a multimodal combined single-task long short-term memory (LSTM) regressor with an input sequence length of 2 data points (2 visits, 6 months apart) augmented with a pretrained Neural Network Estimator to fill in for the missing values. 

With the advances in machine learning for the diagnosis of Alzheimer’s disease (AD), most studies have focused on either identifying the subject’s status through classification algorithms or on predicting their cognitive scores through regression methods, neglecting the potential association between these two tasks. Motivated by the need to enhance the prospects for early diagnosis along with the ability to predict future disease states, this study proposes a deep neural network based on modality fusion, kernelization, and tensorization that perform multiclass classification and longitudinal regression simultaneously within a unified multitask framework. This relationship between multiclass classification and longitudinal regression is found to boost the efficacy of the final model in dealing with both tasks. Different multimodality scenarios are investigated, and complementary aspects of the multimodal features are exploited to simultaneously delineate the subject’s label and predict related cognitive scores at future timepoints using baseline data. The main intent in this multitask framework is to consolidate the highest accuracy possible in terms of precision, sensitivity, F1 score, and area under the curve (AUC) in the multiclass classification task while maintaining the highest similarity in the MMSE score as measured through the correlation coefficient and the RMSE for all time points under the prediction task, with both tasks, run simultaneously under the same set of hyperparameters. The overall accuracy for multiclass classification of the proposed KTMnet method is 66.85 ± 3.77. The prediction results show an average RMSE of 2.32 ± 0.52 and a correlation of 0.71 ± 5.98 for predicting MMSE throughout the time points. These results are compared to state-of-the-art techniques reported in the literature. A discovery from the multitasking of this consolidated machine learning framework is that a set of hyperparameters that optimize the prediction results may not necessarily be the same as those that would optimize the multiclass classification. In other words, there is a breakpoint beyond which enhancing further the results of one process could lead to the downgrading in accuracy for the other.